GroundSight: Augmenting Vision-Language Models with Grounding Information and De-hallucination

• URL: http://arxiv.org/abs/2509.25669v1
• Date: Tue, 30 Sep 2025 02:09:07 GMT
• Title: GroundSight: Augmenting Vision-Language Models with Grounding Information and De-hallucination
• Abstract summary: We propose a method to improve Visual Question Answering (VQA) with Retrieval-Augmented Generation (RAG)<n>Rather than retrieving information based on the entire image, our approach generates a bounding box around the object most relevant to the question.<n>This reduces background noise, improves alignment between visual and textual cues, and helps mitigate hallucinations.
• Score: 2.1517306866297674
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a method to improve Visual Question Answering (VQA) with Retrieval-Augmented Generation (RAG) by introducing text-grounded object localization. Rather than retrieving information based on the entire image, our approach enables the model to generate a bounding box around the object most relevant to the question, allowing for targeted image cropping and focused retrieval. This reduces background noise, improves alignment between visual and textual cues, and helps mitigate hallucinations. Our RAG method enhances context-aware VQA responses increased the accuracy from 22.19% to 25.64%, with an absolute increase of 3.45 percentage points, compared to the baseline Llama-3.2-Vision-11B agent. We also proposed a de-hallucination method based on question type which can effectively reduce the hallucination rate from 65.79% to 13.88% and improves the truthfulness score.

Related papers

• When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs [33.23214854149651]
  This work identifies a distinct failure mode that previous study overlooked: Attention Distraction (AD)<n>To mitigate this issue, we propose MAD-RAG, a training-free intervention that decouples visual grounding from context integration.<n>Extensive experiments on OK-VQA, E-VQA, and InfoSeek demonstrate that MAD-RAG consistently outperforms existing baselines.
  arXiv  Detail & Related papers  (2026-01-30T21:47:00Z)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs [51.93737995405164]
  Large Vision-Language Models (LVLMs) are susceptible to hallucinations.<n>We introduce a novel Conditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy.<n>We show that the proposed method significantly reduces hallucinations in LVLMs while preserving decoding efficiency.
  arXiv  Detail & Related papers  (2025-05-26T08:36:10Z)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling [78.78822033285938]
  Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations.<n>In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification.
  arXiv  Detail & Related papers  (2025-04-17T17:59:22Z)
• Mitigating Low-Level Visual Hallucinations Requires Self-Awareness: Database, Model and Training Strategy [53.07517728420411]
  We introduce the first instruction database specifically focused on hallucinations in low-level vision tasks.<n>We propose the Self-Awareness Failure Elimination (SAFEQA) model to improve the perception and comprehension abilities of the model in low-level vision tasks.<n>We conduct comprehensive experiments on low-level vision tasks, with the results demonstrating that our proposed method significantly enhances self-awareness of the model in these tasks and reduces hallucinations.
  arXiv  Detail & Related papers  (2025-03-26T16:05:01Z)
• Feedback-Driven Vision-Language Alignment with Minimal Human Supervision [11.012355590697064]
  We introduce SVP (Sampling-based Visual Projection), a novel framework that enhances vision-language alignment without relying on manually curated text-image pairs or preference annotation.<n>We evaluate our approach across six key areas: captioning, referring, visual question answering, multitasking, hallucination control, and object recall.<n>Results demonstrate significant improvements, including a 14 % average improvement in captioning tasks, up to 12 % increase in object recall, and significantly reduced hallucinations.
  arXiv  Detail & Related papers  (2025-01-08T15:32:12Z)
• Multi-Modal Hallucination Control by Visual Information Grounding [121.6983694815504]
  We show that Generative Vision-Language Models (VLMs) are prone to generate plausible-sounding textual answers that are not always grounded in the input image. We introduce Multi-Modal Mutual-Information Decoding (M3ID), a new sampling method for prompt amplification. M3ID amplifies the influence of the reference image over the language prior, hence favoring the generation of tokens with higher mutual information with the visual prompt.
  arXiv  Detail & Related papers  (2024-03-20T22:05:18Z)
• Improved Visual Grounding through Self-Consistent Explanations [58.51131933246332]
  We propose a strategy for augmenting existing text-image datasets with paraphrases using a large language model. SelfEQ is a weakly-supervised strategy on visual explanation maps for paraphrases that encourages self-consistency.
  arXiv  Detail & Related papers  (2023-12-07T18:59:22Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.