Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization

• URL: http://arxiv.org/abs/2605.27906v1
• Date: Wed, 27 May 2026 03:27:23 GMT
• Title: Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization
• Abstract summary: Multimodal Large Reasoning Models introduce the reasoning paradigm, demonstrating strong capabilities on complex vision-language tasks.<n>Existing training-based methods typically mitigate hallucinations through response-level direct preference optimization (DPO), where the Chain-of-Thought (CoT) and the final answer are treated as a monolithic output and optimized jointly.<n>We derive Reasoning-Conditioned Direct Preference Optimization (RC-DPO), which models the CoT as a condition for answer generation and contrasts the preference for the same preferred answer under different CoT conditions, promoting answer-supportive reasoning chain alignment.
• Score: 59.20570719781289
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Reasoning Models introduce the reasoning paradigm, demonstrating strong capabilities on complex vision-language tasks. However, they still suffer from severe hallucinations. Existing training-based methods typically mitigate hallucinations through response-level direct preference optimization (DPO), where the Chain-of-Thought (CoT) and the final answer are treated as a monolithic output and optimized jointly. We reveal that this formulation performs similarly to answer-only optimization, suggesting that it primarily learns answer-level preference, while leaving CoT-level supervision insufficiently exploited. To address this issue, we explicitly formulate a CoT-oriented preference term and derive Reasoning-Conditioned Direct Preference Optimization (RC-DPO), which models the CoT as a condition for answer generation and contrasts the preference for the same preferred answer under different CoT conditions, promoting answer-supportive reasoning chain alignment. To further improve optimization, we introduce a reasoning-enhanced preference data generation strategy that employs Monte Carlo Tree Search to discover visually grounded and logically consistent CoTs as positive samples, and attention-guided CoT token pruning to construct negative ones. Extensive experiments across various models and benchmarks show that RC-DPO effectively mitigates hallucinations and improves the reliability of the multimodal reasoning process.

Related papers

• WhisperRec: Latent Reasoning for Efficient Foundation Recommendation Models [51.88075626243057]
  We propose WhisperRec, an efficient latent reasoning framework for foundation recommendation models (FRMs)<n>WhisperRec compresses teacher-generated Chain-of-Thought (CoT) into learnable latent reasoning tokens.<n>We show that WhisperRec consistently outperforms explicit-CoT methods and conventional baselines.
  arXiv  Detail & Related papers  (2026-07-29T08:48:35Z)
• LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models [34.349722314481824]
  Group Relative Policy Optimization is prized for its efficacy in foregoing an explicit value-critic.<n>The method's reliance on a monolithic statistical baseline, such as the group mean, collapses the relational topology of the trajectory space into a single scalar.<n>We introduce a novel framework, Lambda Policy Optimization (LambdaPO), that addresses this information-theoretic bottleneck.
  arXiv  Detail & Related papers  (2026-05-19T06:10:24Z)
• DiffuReason: Bridging Latent Reasoning and Generative Refinement for Sequential Recommendation [20.756497463882763]
  We propose DiffuReason, a unified "Think-then-Diffuse" framework for sequential recommendation.<n>It integrates multi-step Thinking Tokens for latent reasoning, diffusion-based refinement for denoising intermediate representations, and end-to-end Group Relative Policy Optimization.<n>Experiments on four benchmarks demonstrate that DiffuReason consistently improves diverse backbone architectures.
  arXiv  Detail & Related papers  (2026-02-10T12:55:30Z)
• Seeing Through the Chain: Mitigate Hallucination in Multimodal Reasoning Models via CoT Compression and Contrastive Preference Optimization [78.94590726578014]
  multimodal reasoning models (MLRMs) remain prone to hallucinations, and effective solutions are still underexplored.<n>We propose C3PO, a training-based mitigation framework comprising textbfCompression and textbfPreference textbfOptimization.
  arXiv  Detail & Related papers  (2026-02-03T11:00:55Z)
• EntroCoT: Enhancing Chain-of-Thought via Adaptive Entropy-Guided Segmentation [18.606842425858]
  Chain-of-Thought (CoT) prompting has significantly enhanced the mathematical reasoning capabilities of Large Language Models.<n>Existing fine-tuning datasets frequently suffer from the "answer right but reasoning wrong" probelm.<n>This paper proposes EntroCoT, a unified framework for automatically identifying and refining low-quality CoT supervision traces.
  arXiv  Detail & Related papers  (2026-01-07T10:02:27Z)
• Reasoning Pattern Alignment Merging for Adaptive Reasoning [48.347817456299104]
  Reasoning Pattern Alignment Merging (RPAM)<n>RPAM is a layer-wise model merging framework based on feature alignment to facilitate query-adaptive reasoning.<n> Experiments on seven widely used reasoning benchmarks show that RPAM substantially reduces inference cost while maintaining strong performance.
  arXiv  Detail & Related papers  (2026-01-07T01:36:39Z)
• Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendation [60.33386541343322]
  We propose a Multimodal Large Language Models framework that integrates Hardness-aware and Noise-regularized preference optimization for Recommendation (HaNoRec)<n>Specifically, HaNoRec dynamically adjusts optimization weights based on both the estimated hardness of each training sample and the policy model's real-time responsiveness.
  arXiv  Detail & Related papers  (2025-11-24T04:10:46Z)
• Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning [31.727984223052648]
  This paper proposes UnifiedReward-Think, the first unified multimodal CoT-based reward model.<n>We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o.<n>We then prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks.
  arXiv  Detail & Related papers  (2025-05-06T08:46:41Z)
• In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinement [71.60563181678323]
  Large language models (LLMs) have achieved great success across diverse tasks, and fine-tuning is sometimes needed to further enhance generation quality.<n>To handle these challenges, a direct solution is to generate high-confidence'' data from unsupervised downstream tasks.<n>We propose a novel approach, pseudo-supervised demonstrations aligned prompt optimization (PAPO) algorithm, which jointly refines both the prompt and the overall pseudo-supervision.
  arXiv  Detail & Related papers  (2024-10-04T03:39:28Z)
• Step-level Value Preference Optimization for Mathematical Reasoning [6.318873143509028]
  We introduce a novel algorithm called Step-level Value Preference Optimization (SVPO) Our method achieves state-of-the-art performance on both in-domain and out-of-domain mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2024-06-16T09:06:17Z)
• Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
  We introduce a novel closed-form formulation for direct preference optimization using multiple reference models. The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models. Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
  arXiv  Detail & Related papers  (2024-05-26T00:29:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.