From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception

• URL: http://arxiv.org/abs/2604.12508v1
• Date: Tue, 14 Apr 2026 09:32:13 GMT
• Title: From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception
• Abstract summary: Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general visual understanding.<n>They frequently falter in fine-grained perception tasks that require identifying tiny objects or discerning subtle visual relationships.<n>We attribute this limitation to Visual Attenuation: a phenomenon where sparse fine-grained visual signals are prematurely suppressed or diluted by dominant textual tokens during network propagation.<n>Existing input-centric solutions fail to fundamentally reverse this intrinsic mechanism of information loss.<n>We propose the Variational Information Flow (VIF) framework to address this challenge.
• Score: 12.4823697158657
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general visual understanding, they frequently falter in fine-grained perception tasks that require identifying tiny objects or discerning subtle visual relationships. We attribute this limitation to Visual Attenuation: a phenomenon where sparse fine-grained visual signals are prematurely suppressed or diluted by dominant textual tokens during network propagation, resulting in a "loss of focus" during the deep-level decision-making process. Existing input-centric solutions fail to fundamentally reverse this intrinsic mechanism of information loss. To address this challenge, we propose the Variational Information Flow (VIF) framework. Adopting a probabilistic perspective, VIF leverages a Conditional Variational Autoencoder (CVAE) to model the visual saliency relevant to the question-answer pair as a latent distribution. As a plug-and-play module, VIF can be integrated into existing architectures. Extensive evaluations across diverse benchmarks, covering General VQA, fine-grained perception, and visual grounding, demonstrate that VIF yields competitive improvements over previous methods, validating its effectiveness in enhancing the fine-grained perception of MLLMs.

Related papers

• Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models [58.9387276447485]
  Vision Inference Former (VIF) is a lightweight architectural module that establishes a bridge between pure visual representations and the model's output space.<n>We conduct experiments on 14 benchmark tasks covering general reasoning, OCR, table understanding, vision-centric evaluation, and hallucination.
  arXiv  Detail & Related papers  (2026-05-18T10:04:22Z)
• Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs [66.15429821530503]
  Persistent Visual Memory is a lightweight learnable module designed to strengthen sustained, on-demand access to visual evidence.<n>Experiments on Qwen3-VL models demonstrate that PVM brings notable improvements with negligible parameter overhead.<n>In-depth analysis reveals that PVM shows improved robustness in longer generations and accelerates internal prediction convergence.
  arXiv  Detail & Related papers  (2026-05-01T17:54:37Z)
• Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification [55.357038267439684]
  Visual Re-Examination (VRE) is a self-evolving training framework that enables MLLMs to autonomously perform visual introspection during reasoning without additional visual inputs.<n>VRE consistently improves reasoning accuracy and perceptual reliability, while substantially reducing hallucinations, especially in long-chain settings.
  arXiv  Detail & Related papers  (2026-03-27T12:22:13Z)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models [84.94288033791346]
  We conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs.<n>We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation.<n>We propose Predictive Regularization to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM's internal representations.
  arXiv  Detail & Related papers  (2026-03-21T13:10:37Z)
• Rethinking VLMs for Image Forgery Detection and Localization [55.32700985102152]
  We study how to fully leverage vision-language models (VLMs) to assist the image forgery detection and localization (IFDL) task.<n>Building on these findings, we propose a new IFDL pipeline named IFDL-VLM.<n>The experimental results show that we consistently achieve new state-of-the-art performance in detection, localization, and interpretability.
  arXiv  Detail & Related papers  (2026-03-13T12:21:31Z)
• V-ITI: Mitigating Hallucinations in Multimodal Large Language Models via Visual Inference-Time Intervention [39.81171248046778]
  Multimodal Large Language Models (MLLMs) excel in numerous vision-language tasks yet suffer from hallucinations.<n>We propose V-ITI, a lightweight visual inference-time intervention framework integrating a Visual Neglect Detector.<n>Experiments show V-ITI consistently mitigates vision-related hallucinations while preserving general task performance.
  arXiv  Detail & Related papers  (2025-12-03T08:03:54Z)
• Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation [27.62798534410331]
  We introduce Fine-grained Cross-modal Causal Tracing (FCCT) framework, which systematically quantifies the causal effects on visual object perception.<n>FCCT conducts fine-grained analysis covering the full range of visual and textual tokens, three core model components including multi-head self-attention (MHSA), feed-forward networks (FFNs), and hidden states, across all decoder layers.<n>Our analysis is the first to demonstrate that MHSAs of the last token in middle layers play a critical role in aggregating cross-modal information, while FFNs exhibit a three-stage hierarchical progression for the storage
  arXiv  Detail & Related papers  (2025-11-08T08:37:26Z)
• HAMLET-FFD: Hierarchical Adaptive Multi-modal Learning Embeddings Transformation for Face Forgery Detection [6.060036926093259]
  HAMLET-FFD is a cross-domain generalization framework for face forgery detection.<n>It integrates visual evidence with conceptual cues, emulating expert forensic analysis.<n>By design, HAMLET-FFD freezes all pretrained parameters, serving as an external plugin.
  arXiv  Detail & Related papers  (2025-07-28T15:09:52Z)
• GLIMPSE: Holistic Cross-Modal Explainability for Large Vision-Language Models [0.0]
  We introduce GLIMPSE, a model-agnostic framework that jointly attributes LVLM outputs to the most relevant visual evidence and textual signals.<n>GLIMPSE fuses gradient-weighted attention, adaptive layer propagation, and relevance-weighted token aggregation to produce holistic response-level heat maps.<n>We demonstrate an analytic approach to uncover fine-grained insights into LVLM cross-modal attribution, trace reasoning dynamics, analyze systematic misalignment, diagnose hallucination and bias, and ensure transparency.
  arXiv  Detail & Related papers  (2025-06-23T18:00:04Z)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding [5.976839106353883]
  SECOND: Selective and Contrastive Decoding is a novel approach that enables Vision-Language Models to leverage multi-scale visual information with an object-centric manner.<n> SECOND significantly reduces perceptual hallucinations and outperforms a wide range of benchmarks.
  arXiv  Detail & Related papers  (2025-06-10T02:55:38Z)
• Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis [21.869968563545736]
  We define implicit visual misunderstanding (IVM), where MLLMs provide correct answers without fully comprehending the visual input.<n>We introduce a scale-agnostic metric, textitattention accuracy, and a novel benchmark for quantifying IVMs.<n>We extend our approach to finer granularities and demonstrate its effectiveness in unimodal scenarios.
  arXiv  Detail & Related papers  (2025-05-15T17:52:40Z)
• Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descent [72.1517476116743]
  Recent MLLMs have shown emerging visual understanding and reasoning abilities after being pre-trained on large-scale multimodal datasets.<n>Existing approaches, such as direct fine-tuning and continual learning methods, fail to explicitly address this issue.<n>We introduce a novel perspective leveraging effective rank to quantify the degradation of visual representation forgetting.<n>We propose a modality-decoupled gradient descent (MDGD) method that regulates gradient updates to maintain the effective rank of visual representations.
  arXiv  Detail & Related papers  (2025-02-17T12:26:34Z)
• Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding [66.06337890279839]
  Large vision-language models (LVLMs) have shown remarkable capabilities in visual-language understanding for downstream multi-modal tasks.<n>LVLMs still suffer from generating hallucinations in complex generation tasks, leading to inconsistencies between visual inputs and generated content.<n>We propose an Inter-Modality Correlation Decoding (IMCCD) method to mitigate hallucinations in LVLMs in a training-free manner.
  arXiv  Detail & Related papers  (2025-01-03T17:56:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.