HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models

• URL: http://arxiv.org/abs/2608.02124v1
• Date: Mon, 03 Aug 2026 12:15:33 GMT
• Title: HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models
• Abstract summary: Vision-language models (VLMs) remain unreliable when predictions require fine-grained visual evidence.<n>We identify a previously overlooked cause: spectral response rigidity.<n>We propose HAFI-VLM, which introduces a task-conditioned frequency pathway while preserving the pretrained semantic representation.
• Score: 9.926121668688772
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) remain unreliable when predictions require fine-grained visual evidence. We identify a previously overlooked cause: spectral response rigidity. Despite substantial frequency variation across images and tasks, pretrained vision encoders exhibit persistent, encoder-specific layerwise spectral profiles that change only marginally under downstream fine-tuning. Since pretrained vision encoders only receive images, they cannot adapt spectral extraction to the evidence required by the current query. We therefore propose HAFI-VLM, which introduces a task-conditioned frequency pathway while preserving the pretrained semantic representation. Hierarchical Adaptive Frequency Injection (HAFI) retrieves complementary low-, mid-, and high-frequency evidence at multiple encoder depths using text-modulated, spatially aligned cross-attention. A Visual Enrichment Layer Adapter further recalibrates shallow LLM attention to effectively utilize the enriched visual tokens. Experiments on LLaVA-1.5 and Qwen2.5-VL demonstrate consistent improvements in general VQA, text-rich understanding, and hallucination robustness, outperforming representation-level enhancement methods and most resolution- or cropping-based approaches without additional high-resolution encoding. Mechanistic analyses show that HAFI restores task-dependent spectral allocation while retaining semantic attention, establishing frequency enrichment as a distinct and effective route for improving VLM perception.

Related papers

• Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering [2.5820404110321413]
  Medical Visual Question Answering (VQA) requires aligning subtle visual evidence, including lesion texture, boundary sharpness, and diffuse density changes, with clinical language.<n>Existing multimodal fusion approaches operating in the spatial domain may not fully exploit complementary frequency information present in visual and textual representations.<n>We introduce a dual-branch frequency-domain fusion module that conditions spectral filtering on the input question, enabling adaptive selection of global low-frequency structure and fine-grained high-frequency detail before reconstructing the spatial representation for answer generation.
  arXiv  Detail & Related papers  (2026-08-08T19:36:23Z)
• TRUST: Efficient Abdominal Trauma Recognition via Image-to-Ultrasound-Video Transfer Learning [9.697220089288564]
  We propose a scan-aware PEIVTL framework that explicitly models fine-temporal variations to enable reliable ultrasound video understanding.<n>Experiments on in-house ultrasound trauma datasets demonstrate that TRUST outperforms state-of-the-art methods by 9.63% with superior computational efficiency.
  arXiv  Detail & Related papers  (2026-06-26T07:06:34Z)
• SpecPL: Disentangling Spectral Granularity for Prompt Learning [11.908475229249161]
  We introduce Disentangling Spectral Granularity for Prompt Learning (SpecPL), which approaches prompt learning from a novel spectral perspective.<n> Specifically, we leverage a frozen VAE to decompose visual signals into semantic low-frequency bands and granular high-frequency details.<n>Experiments on 11 benchmarks demonstrate competitive state-of-the-art performance, achieving a new performance ceiling of 81.51% harmonic-mean accuracy.
  arXiv  Detail & Related papers  (2026-05-06T05:13:40Z)
• Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs [66.15429821530503]
  Persistent Visual Memory is a lightweight learnable module designed to strengthen sustained, on-demand access to visual evidence.<n>Experiments on Qwen3-VL models demonstrate that PVM brings notable improvements with negligible parameter overhead.<n>In-depth analysis reveals that PVM shows improved robustness in longer generations and accelerates internal prediction convergence.
  arXiv  Detail & Related papers  (2026-05-01T17:54:37Z)
• From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception [12.4823697158657]
  Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general visual understanding.<n>They frequently falter in fine-grained perception tasks that require identifying tiny objects or discerning subtle visual relationships.<n>We attribute this limitation to Visual Attenuation: a phenomenon where sparse fine-grained visual signals are prematurely suppressed or diluted by dominant textual tokens during network propagation.<n>Existing input-centric solutions fail to fundamentally reverse this intrinsic mechanism of information loss.<n>We propose the Variational Information Flow (VIF) framework to address this challenge.
  arXiv  Detail & Related papers  (2026-04-14T09:32:13Z)
• Rethinking VLMs for Image Forgery Detection and Localization [55.32700985102152]
  We study how to fully leverage vision-language models (VLMs) to assist the image forgery detection and localization (IFDL) task.<n>Building on these findings, we propose a new IFDL pipeline named IFDL-VLM.<n>The experimental results show that we consistently achieve new state-of-the-art performance in detection, localization, and interpretability.
  arXiv  Detail & Related papers  (2026-03-13T12:21:31Z)
• High-Fidelity Text-to-Image Generation from Pre-Trained Vision-Language Models via Distribution-Conditioned Diffusion Decoding [64.13126192228604]
  We propose a diffusion-based decoding framework that enhances image fidelity by training only a diffusion decoder on the output image-token logits.<n>A lightweight Logit aligns training-time proxy logits from the VQ-VAE encoder with VLM-generated logits, mitigating the train-inference gap.<n>Our method consistently improves visual fidelity for both VQ-VAE reconstructions and text-to-image generations from VLM-predicted tokens.
  arXiv  Detail & Related papers  (2026-03-11T07:02:48Z)
• Suppressing VLM Hallucinations with Spectral Representation Filtering [49.52703800684483]
  Vision-language models (VLMs) frequently produce hallucinations in the form of descriptions of objects, attributes, or relations that do not exist in the image.<n>We introduce Spectral Representation Filtering (SRF), a lightweight, training-free method to suppress such hallucinations by analyzing and correcting the covariance structure of the model's representations.
  arXiv  Detail & Related papers  (2025-11-15T13:49:27Z)
• On the Reliability of Vision-Language Models Under Adversarial Frequency-Domain Perturbations [53.611451075703314]
  Vision-Language Models (VLMs) are increasingly used as perceptual modules for visual content reasoning.<n>We show how these feature transformations undermine authenticity/DeepFake detection and automated image captioning tasks.
  arXiv  Detail & Related papers  (2025-07-30T05:41:29Z)
• Adaptive Frequency Learning in Two-branch Face Forgery Detection [66.91715092251258]
  We propose Adaptively learn Frequency information in the two-branch Detection framework, dubbed AFD. We liberate our network from the fixed frequency transforms, and achieve better performance with our data- and task-dependent transform layers.
  arXiv  Detail & Related papers  (2022-03-27T14:25:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.