MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs

• URL: http://arxiv.org/abs/2508.10264v2
• Date: Mon, 13 Oct 2025 09:52:34 GMT
• Title: MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs
• Abstract summary: Multi-Region Fusion Decoding (MRFD) is a training-free decoding method that improves factual grounding by modeling inter-region consistency.<n>MRFD identifies salient regions using cross-attention, generates initial responses for each, and computes reliability weights based on Jensen-Shannon Divergence.
• Score: 57.931366749890906
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (LVLMs) have shown strong performance across multimodal tasks. However, they often produce hallucinations -- text that is inconsistent with visual input, due to the limited ability to verify information in different regions of the image. To address this, we propose Multi-Region Fusion Decoding (MRFD), a training-free decoding method that improves factual grounding by modeling inter-region consistency. MRFD identifies salient regions using cross-attention, generates initial responses for each, and computes reliability weights based on Jensen-Shannon Divergence (JSD) among the responses. These weights guide a consistency-aware fusion of per-region predictions, using region-aware prompts inspired by Chain-of-Thought reasoning. Experiments across multiple LVLMs and benchmarks show that MRFD significantly reduces hallucinations and improves response factuality without requiring model updates.

Related papers

• Unified Hallucination Fuzzing for Multimodal Large Language Models [44.4785399881315]
  Hallucination remains a persistent challenge for Multimodal Large Language Models (MLLMs)<n>Existing evaluations, predominantly based on static benchmarks, suffer from narrow taxonomical coverage and rapid performance saturation.<n>We present a systematic evaluation framework integrating a comprehensive benchmark with self-evolving stress testing.
  arXiv  Detail & Related papers  (2026-07-15T17:57:37Z)
• MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs [0.0]
  We propose a Multimodal Adaptive Few-Shot Prompting (MAF) framework for sentiment analysis.<n>MAF retrieves and integrates query-relevant demonstrations to elicit the sentiment reasoning capabilities of MLLMs in a context-sensitive manner.<n>Experiments on public benchmark datasets demonstrate that MAF achieves substantial and consistent performance improvements.
  arXiv  Detail & Related papers  (2026-06-14T09:16:36Z)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation [51.743225614196774]
  Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning.<n>They remain vulnerable to hallucination, where generated content deviates from visual evidence.<n>Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding.<n>We propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs.
  arXiv  Detail & Related papers  (2026-02-27T14:18:51Z)
• Do MLLMs Really See It: Reinforcing Visual Attention in Multimodal LLMs [55.61018839017648]
  Chain-of-thought (CoT) reasoning has substantially improved multimodal large language models (MLLMs) on complex reasoning tasks.<n>Existing approaches largely rely on long textual reasoning trajectories and provide limited mechanisms for learning stable visual attention policies.<n>We propose SAYO, a visual reasoning model trained with a reinforcement learning framework that introduces a region-level visual attention-based reward.
  arXiv  Detail & Related papers  (2026-02-09T03:33:23Z)
• PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Models [43.767942065379366]
  Sycophancy is a tendency of AI models to agree with user input at the expense of factual accuracy or in contradiction of visual evidence.<n>We introduce a comprehensive evaluation benchmark, textitPENDULUM, comprising approximately 2,000 human-curated Visual Question Answering pairs.<n>We observe substantial variability in model robustness and a pronounced susceptibility to sycophantic and hallucinatory behavior.
  arXiv  Detail & Related papers  (2025-12-22T12:49:12Z)
• Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding [41.828387997311474]
  Large Vision-Language Models (LVLMs) have recently achieved impressive results in multimodal tasks such as image captioning and visual question answering.<n>They remain prone to object hallucination -- generating descriptions of nonexistent or misidentified objects.<n>We propose Adaptive Token Ensemble Decoding (ATED), a training-free, token-level ensemble framework that mitigates hallucination by aggregating predictions from multiple LVLMs during inference.
  arXiv  Detail & Related papers  (2025-10-21T06:11:24Z)
• Beyond ROUGE: N-Gram Subspace Features for LLM Hallucination Detection [5.0106565473767075]
  Large Language Models (LLMs) have demonstrated effectiveness across a wide variety of tasks involving natural language.<n>A fundamental problem of hallucinations still plagues these models, limiting their trustworthiness in generating consistent, truthful information.<n>We propose a novel approach inspired by ROUGE that constructs an N-Gram frequency tensor from LLM-generated text.<n>This tensor captures richer semantic structure by encoding co-occurrence patterns, enabling better differentiation between factual and hallucinated content.
  arXiv  Detail & Related papers  (2025-09-03T18:52:24Z)
• ReLoop: "Seeing Twice and Thinking Backwards" via Closed-loop Training to Mitigate Hallucinations in Multimodal understanding [0.053801353100098995]
  Multimodal Large Language Models (MLLMs) have achieved remarkable progress in open-ended visual question answering.<n>MLLMs remain vulnerable to hallucinations, posing a critical challenge to the reliability and factual consistency.<n>We propose ReLoop, a unified closed-loop training framework that encourages multimodal consistency for cross-modal understanding.
  arXiv  Detail & Related papers  (2025-07-07T12:40:48Z)
• HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Models [11.826832299262199]
  HalluRNN enhances model stability through recurrent cross-layer reasoning.<n>By fine-tuning only the DG-DPU module, HalluRNN achieves strong and robust performance across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-06-21T04:56:55Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding [5.71478837100808]
  Large Vision-Language Models (LVLMs) generate descriptions that inaccurately reflect the visual content by including nonexistent objects or misrepresenting existing ones.<n>We propose Ensemble Decoding (ED), a novel strategy that splits the input image into sub-images and combines logit distributions by assigning weights through the attention map.<n>Our proposed method achieves state-of-the-art performance, validating the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-05-23T06:35:43Z)
• Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding [66.06337890279839]
  Large vision-language models (LVLMs) have shown remarkable capabilities in visual-language understanding for downstream multi-modal tasks.<n>LVLMs still suffer from generating hallucinations in complex generation tasks, leading to inconsistencies between visual inputs and generated content.<n>We propose an Inter-Modality Correlation Decoding (IMCCD) method to mitigate hallucinations in LVLMs in a training-free manner.
  arXiv  Detail & Related papers  (2025-01-03T17:56:28Z)
• CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models [16.91226496250909]
  We break down multi-modal understanding into two stages, from Coarse to Fine.<n>In the first stage, we prompt the MLLM to locate the approximate area of the answer.<n>In the second stage, we further enhance the model's focus on relevant areas through visual prompt engineering.
  arXiv  Detail & Related papers  (2024-12-22T05:42:40Z)
• Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs [3.8318712731382054]
  We introduce ContextualLens, a refined method that leverages contextual token embeddings from middle layers of LMMs.<n>This approach significantly improves hallucination detection and grounding across diverse categories, including actions and OCR.<n>Our contributions pave the way for more reliable and interpretable multimodal models.
  arXiv  Detail & Related papers  (2024-11-28T14:47:55Z)
• MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild [81.32127423981426]
  Multimodal emotion recognition based on audio and video data is important for real-world applications. Recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. We propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders.
  arXiv  Detail & Related papers  (2024-04-13T13:39:26Z)
• Learning to Fuse Monocular and Multi-view Cues for Multi-frame Depth Estimation in Dynamic Scenes [51.20150148066458]
  We propose a novel method to learn to fuse the multi-view and monocular cues encoded as volumes without needing the generalizationally crafted masks. Experiments on real-world datasets prove the significant effectiveness and ability of the proposed method.
  arXiv  Detail & Related papers  (2023-04-18T13:55:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.