Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization

• URL: http://arxiv.org/abs/2502.13146v1
• Date: Tue, 18 Feb 2025 18:59:57 GMT
• Title: Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
• Authors: Shuo Xing, Yuping Wang, Peiran Li, Ruizheng Bai, Yueqi Wang, Chengxuan Qian, Huaxiu Yao, Zhengzhong Tu,
• Abstract summary: Large Vision Language Models (VLMs) are prone to significant hallucinations, particularly in the form of cross-modal inconsistencies. We introduce Re-Align, a novel alignment framework that leverages image retrieval to construct a dual-preference dataset. We also introduce rDPO, an extension of the standard direct preference optimization that incorporates an additional visual preference objective during fine-tuning.
• Score: 19.37373012848517
• License:
• Abstract: The emergence of large Vision Language Models (VLMs) has broadened the scope and capabilities of single-modal Large Language Models (LLMs) by integrating visual modalities, thereby unlocking transformative cross-modal applications in a variety of real-world scenarios. Despite their impressive performance, VLMs are prone to significant hallucinations, particularly in the form of cross-modal inconsistencies. Building on the success of Reinforcement Learning from Human Feedback (RLHF) in aligning LLMs, recent advancements have focused on applying direct preference optimization (DPO) on carefully curated datasets to mitigate these issues. Yet, such approaches typically introduce preference signals in a brute-force manner, neglecting the crucial role of visual information in the alignment process. In this paper, we introduce Re-Align, a novel alignment framework that leverages image retrieval to construct a dual-preference dataset, effectively incorporating both textual and visual preference signals. We further introduce rDPO, an extension of the standard direct preference optimization that incorporates an additional visual preference objective during fine-tuning. Our experimental results demonstrate that Re-Align not only mitigates hallucinations more effectively than previous methods but also yields significant performance gains in general visual question-answering (VQA) tasks. Moreover, we show that Re-Align maintains robustness and scalability across a wide range of VLM sizes and architectures. This work represents a significant step forward in aligning multimodal LLMs, paving the way for more reliable and effective cross-modal applications. We release all the code in https://github.com/taco-group/Re-Align.

Related papers

• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs [107.21334626890713]
  Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities. We propose a Cross-modal Hierarchical Direct Preference Optimization (CHiP) to address these limitations. We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations.
  arXiv  Detail & Related papers  (2025-01-28T02:05:38Z)
• ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning [38.26304604660713]
  ADEM-VL is an efficient vision-language method that tunes models based on pretrained large language models. Our framework surpasses existing methods by an average accuracy of 0.77% on ScienceQA dataset.
  arXiv  Detail & Related papers  (2024-10-23T11:31:06Z)
• Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment [57.0121616203175]
  We propose FiSAO, a novel self-alignment method that utilizes the model's own visual encoder as a fine-grained verifier to improve vision-language alignment. By leveraging token-level feedback from the vision encoder, FiSAO significantly improves vision-language alignment, even surpassing traditional preference tuning methods that require additional data.
  arXiv  Detail & Related papers  (2024-10-18T03:34:32Z)
• X-VILA: Cross-Modality Alignment for Large Language Model [91.96081978952283]
  X-VILA is an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities. We propose a visual alignment mechanism with a visual embedding highway module to address the problem of visual information loss. X-VILA exhibits proficiency in any-to-any modality conversation, surpassing previous approaches by large margins.
  arXiv  Detail & Related papers  (2024-05-29T17:59:58Z)
• Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement [102.22911097049953]
  Large vision-language models (LVLMs) have achieved impressive results in visual question-answering and reasoning tasks. Existing methods often depend on external models or data, leading to uncontrollable and unstable alignment results. We propose SIMA, a self-improvement framework that enhances visual and language modality alignment without external dependencies.
  arXiv  Detail & Related papers  (2024-05-24T23:09:27Z)
• Calibrated Self-Rewarding Vision Language Models [27.686545023186852]
  Large Vision-Language Models (LVLMs) have made substantial progress by integrating pre-trained large language models (LLMs) and vision models through instruction tuning. LVLMs often exhibit the hallucination phenomenon, where generated text responses appear linguistically plausible but contradict the input image. We propose the Calibrated Self-Rewarding (CSR) approach, which enables the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning.
  arXiv  Detail & Related papers  (2024-05-23T14:30:33Z)
• Debiasing Multimodal Large Language Models [61.6896704217147]
  Large Vision-Language Models (LVLMs) have become indispensable tools in computer vision and natural language processing. Our investigation reveals a noteworthy bias in the generated content, where the output is primarily influenced by the underlying Large Language Models (LLMs) prior to the input image. To rectify these biases and redirect the model's focus toward vision information, we introduce two simple, training-free strategies.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)
• u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model [17.3535277338312]
  u-LLaVA is an innovative unifying multi-task framework that integrates pixel, regional, and global features to refine the perceptual faculties of MLLMs. This work contributes a novel mask-based multi-task dataset comprising 277K samples, crafted to challenge and assess the fine-grained perception capabilities of MLLMs.
  arXiv  Detail & Related papers  (2023-11-09T13:18:27Z)
• mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections [104.14624185375897]
  mPLUG is a new vision-language foundation model for both cross-modal understanding and generation. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, such as image captioning, image-text retrieval, visual grounding and visual question answering.
  arXiv  Detail & Related papers  (2022-05-24T11:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.