Lost in Embeddings: Information Loss in Vision-Language Models

• URL: http://arxiv.org/abs/2509.11986v1
• Date: Mon, 15 Sep 2025 14:38:06 GMT
• Title: Lost in Embeddings: Information Loss in Vision-Language Models
• Abstract summary: Vision--language models often process visual inputs through a pretrained vision encoder, followed by a projection into the language model's embedding space via a connector component.<n>While crucial for modality fusion, the potential information loss induced by this projection step and its direct impact on model capabilities remain understudied.<n>We introduce two complementary approaches to examine and quantify this loss by analyzing the latent representation space.
• Score: 38.8742043080561
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision--language models (VLMs) often process visual inputs through a pretrained vision encoder, followed by a projection into the language model's embedding space via a connector component. While crucial for modality fusion, the potential information loss induced by this projection step and its direct impact on model capabilities remain understudied. We introduce two complementary approaches to examine and quantify this loss by analyzing the latent representation space. First, we evaluate semantic information preservation by analyzing changes in k-nearest neighbor relationships between image representations, before and after projection. Second, we directly measure information loss by reconstructing visual embeddings from the projected representation, localizing loss at an image patch level. Experiments reveal that connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40--60\% post-projection, correlating with degradation in retrieval performance. The patch-level embedding reconstruction provides interpretable insights for model behavior on visually grounded question-answering tasks, finding that areas of high information loss reliably predict instances where models struggle.

Related papers

• Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMs [4.507919939159203]
  We introduce a diagnostic saliency map method tailored for text generation over images using transformer models.<n>Our approach aggregates the language model's attention over the visual tokens across all heads and layers, then maps this attention back onto the vision encoder's patch grid to localise it over the image.
  arXiv  Detail & Related papers  (2026-07-17T16:40:23Z)
• Diagnosing Visual Ignorance in Vision-Language Models [29.2851901986069]
  Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
  arXiv  Detail & Related papers  (2026-06-05T04:16:01Z)
• Metonymy in vision models undermines attention-based interpretability [31.26551387218204]
  Part-based reasoning is a strategy to make a computer vision model directly focus on the object parts that are relevant to the downstream task.<n>We show that modern pretrained vision transformers violate the locality assumption and exhibit a strong intra-object leakage.<n>We then show that this inherently disentangled feature extraction improves attribute-driven part discovery on a variety of tasks.
  arXiv  Detail & Related papers  (2026-05-07T12:14:46Z)
• From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs [50.185593677108436]
  Multimodal Large Language Models (MLLMs) are increasingly applied to pixel-level vision tasks, yet their intrinsic capacity for spatial understanding remains poorly understood.<n>We investigate segmentation capacity through a layerwise linear probing evaluation across the entire MLLM pipeline: vision encoder, adapter, and LLM.
  arXiv  Detail & Related papers  (2026-03-18T00:22:15Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Generative Learning of Differentiable Object Models for Compositional Interpretation of Complex Scenes [0.20718016474717196]
  This study builds on the architecture of the Disentangler of Visual Priors (DVP)<n>The DVP learns to interpret scenes by decomposing the perceived objects into independent visual aspects of shape, size, orientation, and color appearance.
  arXiv  Detail & Related papers  (2025-06-09T20:06:42Z)
• LayerFlow: Layer-wise Exploration of LLM Embeddings using Uncertainty-aware Interlinked Projections [11.252261879736102]
  LayerFlow is a visual analytics workspace that displays embeddings in an interlinked projection design.<n>It communicates the transformation, representation, and interpretation uncertainty.<n>We show the usability of the presented workspace through replication and expert case studies.
  arXiv  Detail & Related papers  (2025-04-09T12:24:58Z)
• Breaking the Frame: Visual Place Recognition by Overlap Prediction [53.17564423756082]
  We propose a novel visual place recognition approach based on overlap prediction, called VOP.<n>VOP proceeds co-visible image sections by obtaining patch-level embeddings using a Vision Transformer backbone.<n>Our approach uses a voting mechanism to assess overlap scores for potential database images.
  arXiv  Detail & Related papers  (2024-06-23T20:00:20Z)
• RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation [50.35403070279804]
  3D occupancy prediction is an emerging task that aims to estimate the occupancy states and semantics of 3D scenes using multi-view images. We propose RadOcc, a Rendering assisted distillation paradigm for 3D Occupancy prediction.
  arXiv  Detail & Related papers  (2023-12-19T03:39:56Z)
• Mapping EEG Signals to Visual Stimuli: A Deep Learning Approach to Match vs. Mismatch Classification [28.186129896907694]
  We propose a "match-vs-mismatch" deep learning model to classify whether a video clip induces excitatory responses in recorded EEG signals. We demonstrate that the proposed model is able to achieve the highest accuracy on unseen subjects. These results have the potential to facilitate the development of neural recording-based video reconstruction.
  arXiv  Detail & Related papers  (2023-09-08T06:37:25Z)
• Unified Visual Relationship Detection with Vision and Language Models [89.77838890788638]
  This work focuses on training a single visual relationship detector predicting over the union of label spaces from multiple datasets. We propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models. Empirical results on both human-object interaction detection and scene-graph generation demonstrate the competitive performance of our model.
  arXiv  Detail & Related papers  (2023-03-16T00:06:28Z)
• Perceptual Grouping in Contrastive Vision-Language Models [59.1542019031645]
  We show how vision-language models are able to understand where objects reside within an image and group together visually related parts of the imagery. We propose a minimal set of modifications that results in models that uniquely learn both semantic and spatial information.
  arXiv  Detail & Related papers  (2022-10-18T17:01:35Z)
• Reconstruction-guided attention improves the robustness and shape processing of neural networks [5.156484100374057]
  We build an iterative encoder-decoder network that generates an object reconstruction and uses it as top-down attentional feedback. Our model shows strong generalization performance against various image perturbations. Our study shows that modeling reconstruction-based feedback endows AI systems with a powerful attention mechanism.
  arXiv  Detail & Related papers  (2022-09-27T18:32:22Z)
• Object Class Aware Video Anomaly Detection through Image Translation [1.2944868613449219]
  This paper proposes a novel two-stream object-aware VAD method that learns the normal appearance and motion patterns through image translation tasks. The results show that, as significant improvements to previous methods, detections by our method are completely explainable and anomalies are localized accurately in the frames.
  arXiv  Detail & Related papers  (2022-05-03T18:04:27Z)
• Spatio-Temporal Graph for Video Captioning with Knowledge Distillation [50.034189314258356]
  We propose a graph model for video captioning that exploits object interactions in space and time. Our model builds interpretable links and is able to provide explicit visual grounding. To avoid correlations caused by the variable number of objects, we propose an object-aware knowledge distillation mechanism.
  arXiv  Detail & Related papers  (2020-03-31T03:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.