Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs

• URL: http://arxiv.org/abs/2604.24395v1
• Date: Mon, 27 Apr 2026 12:22:35 GMT
• Title: Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs
• Abstract summary: Large Vision-Language Models (LVLMs) frequently suffer from hallucinations.<n>Existing preference learning-based approaches largely rely on proprietary models to construct preference datasets.<n>We propose VErified Self-correction DPO (AVES-DPO), a framework that aligns LVLMs using in-distribution data derived from the model's intrinsic knowledge.<n>Our approach employs a consensus-based verification mechanism to diagnose diverse hallucinations and guides the model to self-correct, thereby generating preference pairs strictly compatible with its internal distribution.
• Score: 17.56508975962831
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (LVLMs) frequently suffer from hallucinations. Existing preference learning-based approaches largely rely on proprietary models to construct preference datasets. We identify that this reliance introduces a distributional mismatch between the proprietary and target models that hinders efficient alignment. To address this, we propose Alignment via VErified Self-correction DPO (AVES-DPO), a framework that aligns LVLMs using in-distribution data derived from the model's intrinsic knowledge. Our approach employs a consensus-based verification mechanism to diagnose diverse hallucinations and guides the model to self-correct, thereby generating preference pairs strictly compatible with its internal distribution. Extensive experiments demonstrate that AVES-DPO surpasses existing baselines in hallucination mitigation while requiring only 5.2k samples.

Related papers

• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models [53.15468578562038]
  We propose an Uncertainty-aware Exploratory Direct Preference Optimization (UE-DPO) method for Multimodal Large Language Models (MLLMs)<n>We first quantify the uncertainty from the model's failure to ground token predictions in the given image.<n>We then encourage more learning pressure on visually deficient tokens in preferred samples, and alleviate the over-penalization of beneficial knowledge in dispreferred samples.
  arXiv  Detail & Related papers  (2026-05-06T13:08:12Z)
• Online Self-Calibration Against Hallucination in Vision-Language Models [23.13137973421435]
  Large Vision-Language Models (LVLMs) often suffer from hallucinations, generating descriptions that include visual details absent from the input image.<n>We propose textbfOnline textbfSelf-textbfCAlibtextbfRation (OSCAR), a framework that integrates Monte Carlo Tree Search with a Dual-Granularity Reward Mechanism to construct preference data.
  arXiv  Detail & Related papers  (2026-05-01T01:03:05Z)
• Dual-Teacher Distillation with Subnetwork Rectification for Black-Box Domain Adaptation [69.7036779439312]
  Black box domain adaptation represents a highly practical yet extremely challenging setting.<n>We propose a dual teacher distillation with subnetwork rectification (DDSR) model that exploits specific knowledge embedded in black box source models.<n>Our approach demonstrates consistent improvements over state of the art methods, including those using source data or models.
  arXiv  Detail & Related papers  (2026-03-24T07:54:19Z)
• SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models [36.40093869747498]
  We propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder latent space.<n>SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones.<n>Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.
  arXiv  Detail & Related papers  (2026-03-19T15:28:08Z)
• Did Models Sufficient Learn? Attribution-Guided Training via Subset-Selected Counterfactual Augmentation [61.248535801314375]
  Subset-Selected Counterfactual Augmentation (SS-CA)<n>We develop Counterfactual LIMA to identify minimal spatial region sets whose removal can selectively alter model predictions.<n>Experiments show that SS-CA improves generalization on in-distribution (ID) test data and achieves superior performance on out-of-distribution (OOD) benchmarks.
  arXiv  Detail & Related papers  (2025-11-15T08:39:22Z)
• Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations [73.37711261605271]
  hallucination mitigation methods are mainly based on preference alignment and require external human annotations or auxiliary models for preference data collection.<n>We propose Autonomous Preference Alignment via Self-Injection (APASI), a novel and generalizable method that mitigates hallucinations without external dependencies.<n>APASI leverages the target LVLM to self-inject hallucinations into a generated response, creating a pair of responses with varying preference levels.
  arXiv  Detail & Related papers  (2025-09-14T14:26:53Z)
• OViP: Online Vision-Language Preference Learning for VLM Hallucination [44.14029765850719]
  Large vision-language models (LVLMs) remain vulnerable to hallucination, often generating content misaligned with visual inputs.<n>We propose an Online Vision-language Preference Learning framework that dynamically constructs contrastive training data based on the model's own hallucinations.
  arXiv  Detail & Related papers  (2025-05-21T19:26:09Z)
• Self-NPO: Negative Preference Optimization of Diffusion Models by Simply Learning from Itself without Explicit Preference Annotations [60.143658714894336]
  Diffusion models have demonstrated remarkable success in various visual generation tasks, including image, video, and 3D content generation.<n> Preference optimization (PO) is a prominent and growing area of research that aims to align these models with human preferences.<n>We introduce Self-NPO, a Negative Preference Optimization approach that learns exclusively from the model itself.
  arXiv  Detail & Related papers  (2025-05-17T01:03:46Z)
• Multimodal Preference Data Synthetic Alignment with Reward Model [23.978820500281213]
  We propose a new framework in generating synthetic data using a reward model as a proxy of human preference for effective multimodal alignment with DPO training.<n>Experiment results indicate that integrating selected synthetic data, such as from generative and rewards models can effectively reduce reliance on human-annotated data.
  arXiv  Detail & Related papers  (2024-12-23T09:29:40Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.