PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment

• URL: http://arxiv.org/abs/2608.19598v1
• Date: Thu, 20 Aug 2026 03:35:07 GMT
• Title: PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
• Abstract summary: We propose Perception-Enhanced Alignment DPO (PEA-DPO) to overcome visual insensitivity.<n>PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.<n> Evaluations across three benchmarks using MLLMs of varying scales show that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.
• Score: 34.498065511310195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term visual insensitivity: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely Across-Image Insensitivity and Within-Image Insensitivity. To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO), a framework for multimodal LLMs alignment, which explicitly leverages visual preference signals to overcome visual insensitivity. We further provide a theoretical analysis demonstrating that PEA-DPO provably mitigates both failure modes. Empirical results demonstrate that PEA-DPO enhances sensitivity to visual context while preserving the language modeling capacity of the base model. Evaluations across three hallucination benchmarks using MLLMs of varying scales show that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.

Related papers

• P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization [55.74731799669337]
  Hallucination has recently garnered significant research attention in Large Vision-Language Models (LVLMs)<n>Direct Preference Optimization (DPO) aims to learn directly from the corrected preferences provided by humans.<n>Existing preference pairs are often vision-agnostic and their inherently off-policy nature limits their effectiveness in guiding model learning.<n>We propose Perceptual Processing Direct Preference Optimization (P$2$-DPO), a novel training paradigm in which the model generates and learns from its own preference pairs.
  arXiv  Detail & Related papers  (2026-06-02T09:22:53Z)
• MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization [4.088161686930475]
  We propose Modality-Decoupled Direct Preference Optimization (MoD-DPO) for improving modality grounding in omni LLMs.<n>MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities.<n>Experiments demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines.
  arXiv  Detail & Related papers  (2026-03-03T17:50:24Z)
• Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization [68.64764778089229]
  We propose MISP-DPO, the first framework to incorporate multiple, semantically diverse negative images in multimodal DPO.<n>Our method embeds prompts and candidate images in CLIP space and applies a sparse autoencoder to uncover semantic deviations into interpretable factors.<n>Experiments across five benchmarks demonstrate that MISP-DPO consistently improves multimodal alignment over prior methods.
  arXiv  Detail & Related papers  (2025-09-30T03:24:09Z)
• Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization [69.05600758833471]
  Direct Preference Optimization (DPO) has emerged as an effective approach for mitigating hallucination in Multimodal Large Language Models (MLLMs)<n>We propose a Symmetric Multimodal Preference Optimization (SymMPO) which conducts symmetric preference learning with direct preference supervision (i.e., response pairs)<n>In addition to conventional ordinal preference learning, SymMPO introduces a preference margin consistency loss to quantitatively regulate the preference gap between symmetric preference pairs.
  arXiv  Detail & Related papers  (2025-06-13T12:29:15Z)
• Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs [74.74767980885758]
  We propose Context-to-Cue Direct Preference Optimization (CcDPO), a multi-level preference optimization framework.<n>CcDPO enhances per-image perception in multi-image settings by zooming into visual clues -- from sequential context to local details.<n> Experiments show that CcDPO significantly reduces hallucinations and yields consistent performance gains.
  arXiv  Detail & Related papers  (2025-05-28T14:24:02Z)
• AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization [26.03204301595711]
  We propose an Adaptive Vision-enhanced Preference optimization (AdaViP) that addresses limitations through two key innovations.<n> vision-based preference pair construction integrates multiple visual foundation models to strategically remove key visual elements from the image.<n>AdaViP-7B achieves 93.7% and 96.4% reductions in response-level and mentioned-level hallucination respectively on the Object HalBench.
  arXiv  Detail & Related papers  (2025-04-22T06:19:38Z)
• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs [107.21334626890713]
  Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities.<n>We propose a Cross-modal Hierarchical Direct Preference Optimization (CHiP) to address these limitations.<n>We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations.
  arXiv  Detail & Related papers  (2025-01-28T02:05:38Z)
• mDPO: Conditional Preference Optimization for Multimodal Large Language Models [52.607764280030196]
  Direct preference optimization (DPO) has shown to be an effective method for large language model (LLM) alignment. Recent works have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement. We propose mDPO, a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference.
  arXiv  Detail & Related papers  (2024-06-17T17:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.