Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigation

• URL: http://arxiv.org/abs/2605.25377v1
• Date: Mon, 25 May 2026 03:05:04 GMT
• Title: Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigation
• Abstract summary: Large Vision-Language Models (LVLMs) have advanced multimodal understanding, yet their reliability is limited by hallucinations.<n>We propose Adrial Orthogonal Disentanglement (AOD), a latent geometric framework for mitigating hallucinations.
• Score: 35.94927596542353
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (LVLMs) have advanced multimodal understanding, yet their reliability is limited by hallucination, where generated content conflicts with visual facts. Existing mitigation methods either rely on costly external interventions, such as instruction tuning and retrieval, or use internal mechanisms that remain limited by flawed attention weights and entangled hidden representations. We propose Adversarial Orthogonal Disentanglement (AOD), a latent geometric framework for mitigating LVLM hallucinations. AOD learns a hallucination-related direction through a minimax objective: a classifier concentrates hallucination signals into the projected component, while an adversary removes them from the orthogonal residual space via a Gradient Reversal Layer. The learned direction enables a training-free dual-forward-pass contrastive decoding strategy that suppresses hallucinations while preserving general capabilities. Experiments on three LVLMs across four hallucination and four utility benchmarks show that AOD consistently outperforms strong baselines. It improves POPE accuracy by over 6\% on average, boosts AMBER by 6\%, and maintains strong performance on utility tasks such as MMMU. Further analysis shows robust transfer across datasets, suggesting that AOD captures general hallucination-related biases rather than dataset-specific artifacts. Our source code and datasets are available at https://github.com/Hunter-Wrynn/AOD.

Related papers

• Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment [45.92240827409312]
  Saliency-Driven Perceptual Realignment (SDPR) is a training-free framework that mitigates the degradation of visual awareness throughout inference.<n>Our proposed SDPR is robust against hallucinations due to its holistic alignment of visual awareness across the entire generative trajectory.
  arXiv  Detail & Related papers  (2026-07-18T14:39:16Z)
• VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editing [70.82867621856968]
  Large vision-language models (LVLMs) frequently suffer from Object Hallucination (OH)<n>Recent studies suggest that the hallucination problem may stem from language priors.<n>We propose Visual Contrastive Editing (VCE), a novel post-hoc method that identifies and suppresses hallucinatory tendencies.
  arXiv  Detail & Related papers  (2026-04-21T12:40:07Z)
• Mitigating Multimodal Hallucination via Phase-wise Self-reward [37.441718699530526]
  We introduce a new self-rewarding framework, enabling dynamic hallucination mitigation at inference time without external supervision.<n>textbfPSRD (textbfPhase-wise textbfSelf-textbfReward textbfDecoding) for online hallucination correction guided by phase-wise self-reward signals.
  arXiv  Detail & Related papers  (2026-04-20T09:04:49Z)
• Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction [49.96701537295129]
  Large Vision-Language Models (LVLMs) have achieved remarkable success across cross-modal tasks but remain hindered by hallucinations.<n>Existing methods mitigate hallucinations but often alter generation behavior, resulting in shorter outputs and shifted token distributions.<n>We propose MESA, an effective plug-and-play framework that performs controlled and selective latent intervention for hallucination mitigation.
  arXiv  Detail & Related papers  (2026-04-09T07:31:27Z)
• Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation [86.37623966653688]
  Large vision-language models (LVLMs) have become increasingly strong but remain prone to hallucinations in multimodal tasks.<n>As training these LVLMs to avoid hallucinations becomes prohibitively expensive for larger models, training-free methods offer a cheap and flexible solution to this problem.<n>We propose Kestrel, a training-free framework for LVLM hallucination mitigation that combines an explicit visual-grounding agent with evidence-verified self-refinement mechanism.
  arXiv  Detail & Related papers  (2026-03-17T15:30:47Z)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation [51.743225614196774]
  Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning.<n>They remain vulnerable to hallucination, where generated content deviates from visual evidence.<n>Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding.<n>We propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs.
  arXiv  Detail & Related papers  (2026-02-27T14:18:51Z)
• Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchors [8.089908150148554]
  Multimodal large language models (MLLMs) have achieved remarkable success across diverse vision-language tasks.<n>MLLMs are highly susceptible to hallucinations, producing content that is fluent but inconsistent with visual evidence.<n>We propose a training-free, self-supervised method for hallucination mitigation.
  arXiv  Detail & Related papers  (2025-09-26T07:24:28Z)
• Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations [73.37711261605271]
  hallucination mitigation methods are mainly based on preference alignment and require external human annotations or auxiliary models for preference data collection.<n>We propose Autonomous Preference Alignment via Self-Injection (APASI), a novel and generalizable method that mitigates hallucinations without external dependencies.<n>APASI leverages the target LVLM to self-inject hallucinations into a generated response, creating a pair of responses with varying preference levels.
  arXiv  Detail & Related papers  (2025-09-14T14:26:53Z)
• Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation [38.43656456659151]
  Large vision-language models (LVLMs) have achieved remarkable performance on multimodal tasks.<n>They still suffer from hallucinations, generating text inconsistent with visual input, posing significant risks in real-world applications.<n>We propose Steering LVLMs via SAE Latent Directions (SSL), a plug-and-play method based on SAE-derived latent directions to mitigate hallucinations in LVLMs.
  arXiv  Detail & Related papers  (2025-05-22T02:45:45Z)
• HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data [102.56792377624927]
  hallucinations inherent in machine-generated data remain under-explored. We present a novel hallucination detection and elimination framework, HalluciDoctor, based on the cross-checking paradigm. Our method successfully mitigates 44.6% hallucinations relatively and maintains competitive performance compared to LLaVA.
  arXiv  Detail & Related papers  (2023-11-22T04:52:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.