Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2609.17269v1
• Date: Tue, 15 Sep 2026 14:48:48 GMT
• Title: Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Models
• Abstract summary: In medication assistance, accessible perception, and environmental decision-making, such hallucinations can create real-world safety risks.<n>We propose Semantic-Spatial Agreement Verification (SSAV), a training-free method for verifying object claims.
• Score: 8.031456706175014
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models generate natural-language responses from visual inputs, yet may mention objects absent from an image. In medication assistance, accessible perception, and environmental decision-making, such hallucinations can create real-world safety risks. We propose Semantic-Spatial Agreement Verification (SSAV), a training-free method for verifying object claims. A visually grounded claim should remain stable across semantically equivalent queries and repeatedly localize to the same image region. SSAV aggregates multiple prompts to estimate semantic support and reduce sensitivity to query wording. Query-Induced Regional Verification (QIRV) combines cross-query region persistence, spatial overlap, and relative candidate dominance to identify isolated high responses and dispersed localizations. A geometric mean fuses semantic and spatial evidence, lowering the verification score when either branch lacks support. Experiments on three base models and multiple evaluation protocols show that SSAV effectively mitigates object hallucination. On LLaVA-1.5-7B, accuracy averaged across COCO, A-OKVQA, and GQA improves by 1.81 and 3.17 percentage points under POPE Popular and Adversarial, respectively, while CHAIRs decreases from 49.40% to 32.80%. These results show that cross-query semantic stability and regional consistency provide interpretable external visual evidence for object claims.

Related papers

• MS-MFAD : Multimodal large language models for Face Anti-spoofing Detection [25.077055201475442]
  We propose Multimodal Large Language Models (MFAD) for face anti-spoofing detection.<n>MFAD activates intrinsic reasoning capabilities of Multimodal Large Language Models via a fine-grained pixel-semantic anchoring mechanism.<n>By annotating only 1,000 precise masks per attack category, we generate reasoning evidence chains strictly corresponding to spoofed regions.
  arXiv  Detail & Related papers  (2026-08-18T03:38:59Z)
• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• SASGeo: Stability-Aware Semantic Map Localization for GNSS-Denied UAVs -- A Framework and Synthetic Proof of Concept [0.0]
  sas is a semantic map-localization framework that represents the environment through persistent structures.<n>sas combines semantic alignment, graph evidence, feature stability and geographic distinctiveness, explicit positive/contradictory/unknown observations, and integrity-aware rejection of ambiguous fixes.<n>Preliminary experiment does not validate real-flight navigation; rather, it demonstrates that structured semantic geometry can discriminate locations under controlled cross-view perturbations.
  arXiv  Detail & Related papers  (2026-07-07T16:57:43Z)
• VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence [26.0945130521806]
  Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation.<n>We introduce VISTAQA, a benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering.<n>We introduce GROVE, a metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean.
  arXiv  Detail & Related papers  (2026-05-20T03:44:06Z)
• Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty [70.43119366710778]
  We introduce Agent-BRACE: Agent Belief state Representation via Abstraction and Confidence Estimation.<n>It is a method that decouples an LLM agent into a belief state model and a policy model, jointly optimized via reinforcement learning.<n>It achieves an average absolute improvement of +14.5% across long-horizon, partially observable language environments.
  arXiv  Detail & Related papers  (2026-05-12T02:37:04Z)
• Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs [47.94507630961399]
  We introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time.<n>We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error.<n> Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework.
  arXiv  Detail & Related papers  (2026-03-26T17:53:49Z)
• Deterministic Hallucination Detection in Medical VQA via Confidence-Evidence Bayesian Gain [6.272224314406867]
  Multimodal large language models (MLLMs) have shown strong potential for medical Visual Question Answering (VQA)<n>They remain prone to hallucinations, defined as generating responses that contradict the input image, posing serious risks in clinical settings.<n>Current hallucination detection methods, such as Semantic Entropy (SE) and Vision-Amplified Semantic Entropy (VASE), require 10 to 20 generations per sample together with an external natural language inference model for semantic clustering, making them expensive and difficult to deploy in practice.<n>We propose a deterministic detection method that requires no sampling, no external models, and no task-
  arXiv  Detail & Related papers  (2026-03-23T08:29:53Z)
• Semantic Visual Anomaly Detection and Reasoning in AI-Generated Images [96.43608872116347]
  AnomReason is a large-scale benchmark with structured annotations as quadruple textbfAnomAgent<n>AnomReason and AnomAgent serve as a foundation for measuring and improving the semantic plausibility of AI-generated images.
  arXiv  Detail & Related papers  (2025-10-11T14:09:24Z)
• VOGUE: Guiding Exploration with Visual Uncertainty Improves Multimodal Reasoning [62.09195763860549]
  Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models (LLMs) but struggles with exploration.<n>We introduce $textbfVOGUE (Visual Uncertainty Guided Exploration)$, a novel method that shifts exploration from the output (text) to the input (visual) space.<n>Our work shows that grounding exploration in the inherent uncertainty of visual inputs is an effective strategy for improving multimodal reasoning.
  arXiv  Detail & Related papers  (2025-10-01T20:32:08Z)
• MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs [57.931366749890906]
  Multi-Region Fusion Decoding (MRFD) is a training-free decoding method that improves factual grounding by modeling inter-region consistency.<n>MRFD identifies salient regions using cross-attention, generates initial responses for each, and computes reliability weights based on Jensen-Shannon Divergence.
  arXiv  Detail & Related papers  (2025-08-14T01:17:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.